feat(ltx2): LTX-2.3 LoRA support (map gate_logits + prompt_adaln keys, fix example config to 22b)#425
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
tsvikas
changed the title
tsvikas
force-pushed
the
ltx2-3-lora-support
branch
from
6bf1fac to
02b7d54
Compare
|
Looks good! Tested on 7x-8 with both single LoRA and multi-LoRA weights and output is generated with no LoRA layers being skipped. Regarding your query, we can leave it as log, no need to raise/fail. Please squash your commits |
|
Thanks! |
|
Please squash and force push. I will drop the approval after |
tsvikas
force-pushed
the
ltx2-3-lora-support
branch
from
02b7d54 to
74821fc
Compare
Adds LoRA support for **LTX-2.3 (22b)** in the LTX-2 pipeline by extending the NNX→Diffusers LoRA key-translation map and clarifying the loader's diagnostics. - Map extensions (`loaders/lora_conversion_utils.py`): - `to_gate_logits` for every attention family - `prompt_adaln` / `audio_prompt_adaln` → the model's `prompt_adaln_single` / `audio_prompt_adaln_single`. - Loader logging/diagnostics (`loaders/ltx2_lora_nnx_loader.py`): - a missing LoRA weight name returns early with one clear message. - a missing `transformer` and a missing `connectors` each get their own message. - a LoRA carrying keys outside the two routed prefixes (`diffusion_model.` / `text_embedding_projection.`) is now warned about. - Example config (`configs/ltx2_3_video.yml`): - Point the example `lora_config` at a valid 22b adapter (colorization IC-LoRA) instead of the prior 19b entry.
tsvikas
force-pushed
the
ltx2-3-lora-support
branch
from
74821fc to
0fbf767
Compare
|
squashed. |
|
seems like Merging is blocked from |
|
@tsvikas We have a bot that merges the PR once it's approved internally in Google codebase. I have requested the owners to look into this. |
copybara-service
Bot
merged commit eae38c4
into
AI-Hypercomputer:main
2 participants
Summary
Adds LoRA support for LTX-2.3 (22b) in the LTX-2 pipeline by extending the NNX→Diffusers LoRA key-translation map and clarifying the loader's diagnostics.
loaders/lora_conversion_utils.py):to_gate_logitsfor every attention family: self (attn1), cross (attn2), audio (audio_attn1/audio_attn2), and cross-modal (audio_to_video_attn/video_to_audio_attn).prompt_adaln/audio_prompt_adaln(linear + timestep-embedder) → the model'sprompt_adaln_single/audio_prompt_adaln_single.loaders/ltx2_lora_nnx_loader.py), merge behavior unchanged:transformerand a missingconnectorseach get their own message (the latter had none before).diffusion_model./text_embedding_projection.) is now warned about. Those keys were previously dropped with no notice.configs/ltx2_3_video.yml): point the examplelora_configat the 22b adapter (colorization IC-LoRA) instead of the prior 19b camera-control entry, which is the wrong model size for the 2.3 base.Scope & validation
GEN_EXIT=0. The same generation segfaults at the denoise compile on the v7 host we used — but so does the no-LoRA run there, and the bare config ran clean on a different v7 host, so it's a host-dependent base-pipeline instability (LTX-2.3 pipeline: native SIGSEGVs on tpu7x #423), not the LoRA. (These IC-LoRAs need a vid2vid/reference path for their intended effect, which this repo lacks, so the v6e clip is a valid with-LoRA generation, not a colorization ablation.)Question for maintainers
Today the loader logs any keys that route to no merge target. Would you prefer it to raise instead (fail loudly on a wrong-prefix / zero-coverage load rather than silently no-op)? Happy to add that if you'd like; left as log-only for now to avoid changing behavior without your call.
🤖 Generated with Claude Code